Video frame synthesis using tensor neural networks

ABSTRACT

A method for implementing video frame synthesis using a tensor neural network includes receiving input video data including one or more missing frames, converting the input video data into an input tensor, generating, through tensor completion based on the input tensor, output video data including one or more synthesized frames corresponding to the one or more missing frames by using a transform-based tensor neural network (TTNet) including a plurality of phases implementing a tensor iterative shrinkage thresholding algorithm (ISTA), and obtaining a loss function based on the output video data.

BACKGROUND

The present invention generally relates to artificial intelligence and,more particularly, to video frame synthesis using tensor neuralnetworks.

Video frame synthesis is a task in computer vision that has drawn greatinterest in a wide variety of applications. During video framesynthesis, new video frames are constructed from an existing video.Video frame synthesis can be divided into two categories, video frameinterpolation and video frame prediction. Video frame interpolationconstructs new video frames by recovering missing frames between theexisting video frames, and video frame prediction constructs new videoframes by generating future frames from prior frames. Video framesynthesis techniques have a variety of uses. For example, video framesynthesis techniques can transform standard videos into high-qualityslow-motion videos with higher frame rates and smooth view transitions,compensate for distortions from camera shake in video recording usingvideo stabilization, and mitigate the missing video frame problem inwireless video transmissions using motion smoothing (e.g. in unmannedaerial vehicle (UAV) and virtual reality (VR) applications).

Existing video frame synthesis solutions suffer from one or moredeficiencies. For example, existing neural network-based methods do notexplicitly impose tensor-low rankness to capture spatiotemporalcorrelations of video frames in a high-dimensional space, and do nothave potentially mathematical interpretations. Additionally, compressivesensing-based iterative algorithms generally require handcraftedparameters and a relatively long running time for high-quality videoframe synthesis. For example, some existing optimization algorithms mayrequire more than an hour to process a one minute video (30 frames persecond) with 100 missing frames. Therefore, it would be advantageous toprovide a video synthesis solution that addresses at least theabove-noted problems.

SUMMARY

A method for implementing video frame synthesis using a tensor neuralnetwork includes receiving input video data including one or moremissing frames, converting the input video data into an input tensor,generating, through tensor completion based on the input tensor, outputvideo data including one or more synthesized frames corresponding to theone or more missing frames by using a transform-based tensor neuralnetwork (TTNet) including a plurality of phases implementing a tensoriterative shrinkage thresholding algorithm (ISTA), and obtaining a lossfunction based on the output video data.

In an embodiment of the method, generating the output tensor includes,for a given one of the plurality of phases of the TTNet, updating anintermediate synthesis result in an original domain, transforming theintermediate synthesis result in the original domain into a transformedintermediate synthesis result in a transform domain, applyingsoft-thresholding based on the transformed intermediate synthesis resultto generate synthesized video data in the transform domain, andtransforming the synthesized video data in the transform domain back tothe original domain using an inverse transformation.

In an embodiment of the method, the intermediate synthesis result in theoriginal domain is defined based in part on a video tensor received bythe given one of the plurality of phases and an observation tensor.

In an embodiment of the method, transforming the intermediate synthesisresult in the original domain into the transformed intermediatesynthesis result includes applying a first convolution to theintermediate synthesis result, applying an activation function to anoutput of the first convolution, and applying a second convolution to anoutput of the activation function to generate the transformation. In oneembodiment, the first and second convolutions include two-dimensional(2D) multi-channel convolutions with different kernels, and theactivation function is a rectified linear unit (ReLU).

In an embodiment of the method, applying soft-thresholding based on thetransformed intermediate synthesis result to generate the synthesizedvideo data in the transform domain includes applying a plurality ofsoft-thresholding operations in parallel to each frontal slice of thetransformed intermediate synthesis result, and stacking outputs of theplurality of soft-thresholding operations to generate the synthesizedvideo data in the transform domain.

In an embodiment of the method, transforming the synthesized video datain the transform domain back to the original domain using an inversetransformation includes applying a third convolution to the synthesizedvideo data in the transform domain, applying a second activationfunction to an output of the third convolution, and applying a fourthconvolution to an output of the second activation function. In oneembodiment, the third and fourth convolutions 2D multi-channelconvolutions with different kernels, and the activation function is aReLU.

In an embodiment of the method, transforming the synthesized video datain the transform domain back to the original domain using an inversetransformation includes applying a first convolution to the synthesizedvideo data in the transform domain, applying an activation function toan output of the third convolution, and applying a second convolution toan output of the activation function. In one embodiment, the first andsecond convolutions include 2D multi-channel convolutions with differentkernels, and the activation function is a ReLU.

In an embodiment of the method, the loss function includes a first partthat evaluates accuracy of the one or more synthesized frames, and asecond part that imposes sparsity. More specifically, the loss functionmay be a linear combination of the first and second parts.

A system for implementing video synthesis using a tensor neural networkincludes a memory configured to store program code, and at least oneprocessor device operatively coupled to the memory. The at least oneprocessor device is configured to execute program code stored on thememory device to receive input video data including one or more missingframes, convert the input video data into an input tensor, generate,through tensor completion based on the input tensor, output video dataincluding one or more synthesized frames corresponding to the one ormore missing frames by using a transform-based tensor neural network(TTNet) including a plurality of phases implementing a tensor iterativeshrinkage thresholding algorithm (ISTA), and obtain a loss functionbased on the output video data.

In an embodiment of the system, the at least one processor device isconfigured to generate the output tensor by, for a given one of theplurality of phases of the TTNet, updating an intermediate synthesisresult in an original domain, transforming the intermediate synthesisresult in the original domain into a transformed intermediate synthesisresult in a transform domain, applying soft-thresholding based on thetransformed intermediate synthesis result to generate synthesized videodata in the transform domain, and transforming the synthesized videodata in the transform domain back to the original domain using aninverse transformation.

In an embodiment of the system, the intermediate synthesis result in theoriginal domain is defined based in part on a video tensor received bythe given one of the plurality of phases and an observation tensor.

In an embodiment of the system, the at least one processor device isconfigured to transform the intermediate synthesis result in theoriginal domain into the transformed intermediate synthesis result byapplying a first convolution to the intermediate synthesis result,applying an activation function to an output of the first convolution,and applying a second convolution to an output of the activationfunction to generate the transformation. In one embodiment, the firstand second convolutions include two-dimensional (2D) multi-channelconvolutions with different kernels, and the activation function is arectified linear unit (ReLU).

In an embodiment of the system, the at least one processor device isconfigured to apply soft-thresholding based on the transformedintermediate synthesis result to generate the synthesized video data inthe transform domain by applying a plurality of soft-thresholdingoperations in parallel to each frontal slice of the transformedintermediate synthesis result, and stacking outputs of the plurality ofsoft-thresholding operations to generate the synthesized video data inthe transform domain.

In an embodiment of the system, the at least one processor device isconfigured to transform the synthesized video data in the transformdomain back to the original domain using an inverse transformationincludes applying a first convolution to the synthesized video data inthe transform domain, applying an activation function to an output ofthe third convolution, and applying a second convolution to an output ofthe activation function. In one embodiment, the first and secondconvolutions include 2D multi-channel convolutions with differentkernels, and the activation function is a ReLU.

In an embodiment of the system, the loss function includes a first partthat evaluates accuracy of the one or more synthesized frames, and asecond part that imposes sparsity. More specifically, the loss functionmay be a linear combination of the first and second parts.

These and other features and advantages will become apparent from thefollowing detailed description of illustrative embodiments thereof,which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The following description will provide details of preferred embodimentswith reference to the following figures wherein:

FIG. 1 is a block/flow diagram of a high-level overview of a video framesynthesis system using a transform-based tensor neural network (TTNet),in accordance with an embodiment of the present invention;

FIG. 2 is a block/flow diagram of modules of a phase of the TTNet ofFIG. 1, in accordance with an embodiment of the present invention;

FIG. 3 is a block/flow diagram of an exemplary transform module of thephase of FIG. 2, in accordance with an embodiment of the presentinvention;

FIG. 4 is a block/flow diagram of an exemplary soft-thresholding moduleof the phase of FIG. 2, in accordance with an embodiment of the presentinvention;

FIG. 5 is a block/flow diagram of an exemplary inverse transform moduleof the phase of FIG. 2, in accordance with an embodiment of the presentinvention;

FIG. 6 is exemplary pseudocode depicting a tensor iterative shrinkagethresholding algorithm (ISTA), in accordance with an embodiment of thepresent invention;

FIG. 7 is a block/flow diagram of a system/method for implementing videoframe synthesis using a transform-based tensor neural network (TTNet)implementing a tensor iterative shrinkage thresholding algorithm (ISTA),in accordance with an embodiment of the present invention;

FIG. 8 is a block/flow diagram of a system/method for implementing aninference iteration of the tensor ISTA of FIG. 7, in accordance with anembodiment of the present invention;

FIG. 9 is a block diagram of a processing system, in accordance with anembodiment of the present invention;

FIG. 10 is a block diagram showing an illustrative cloud computingenvironment having one or more cloud computing nodes with which localcomputing devices used by cloud consumers communicate in accordance withone embodiment; and

FIG. 11 is a block diagram showing a set of functional abstractionlayers provided by a cloud computing environment in accordance with oneembodiment.

DETAILED DESCRIPTION

Embodiments of the present invention use a tensor neural network toperform video synthesis. More specifically, the embodiments describedherein can represent video data as a tensor and construct atransform-based tensor neural network (TTNet) by unfolding a tensoriterative shrinkage thresholding algorithm (ISTA) into a multi-phaseneural network. That is, the embodiments described herein cast the videoframe synthesis task as a tensor recovery problem by reconstructing amultiway tensor from a subset of slices (e.g., frontal slices). Due tospatiotemporal correlations of video frames, the embodiments describedherein can solve the tensor recovery problem by performing tensor rankminimization in a transform domain. The embodiments described herein canexploit the low-rankness of video data in the transform domain with arelatively high synthesis speed.

The embodiments described herein can provide improvements over existingvideo synthesis techniques (e.g., interpolation and prediction of videoframes). For example, the embodiments described herein can improve thepeak signal-to-noise ratio (PSNR) of interpolation and prediction of,e.g., 4.13 dB and 4.16 dB. Additional advantages over existing videosynthesis solutions include, for example, motion blurring reduction andartifact avoidance.

Referring now to the drawings in which like numerals represent the sameor similar elements and initially to FIG. 1, a block/flow diagram isprovided showing a high-level overview of a video frame synthesis system100.

As shown, the system 100 includes input video data 110 including aplurality of frames. For example, the input video data 110 can include aframe 112. Although the input video data 110 is shown including threeframes, the input video data 110 can include any suitable number offrames in accordance with the embodiments described herein. Morespecifically, the input video data 110 includes one or more missingframes.

The primary goal of the system 100 is to synthesize the one or moremissing video frames to generate recovered video data (e.g., using videoframe interpolation and/or prediction). To achieve this goal, the system100 further includes a transform-based tensor neural network (TTNet)120.

The following definitions (1)-(5) will be needed to further describe thevideo frame synthesis task performed by the system 100. Given aninvertible linear transform L:

^(1×1×n) ³ →

^(1×1×n) ³ , the tube multiplication of x, y∈

^(1×1×n) ³ , ⊗, can be defined as:

x⊗y=L ⁻¹(L(x)⊙L(y))  (1)

where L⁻¹ is the inverse of L and ⊙ denotes the Hadamard (element-wise)product. The L-product of tensors X∈

^(n) ¹ ^(×n) ² ^(×n) ³ and Y∈

^(n) ² ^(×n) ⁴ ^(×n) ³ , Z=X⊗Y, a tensor of size n₁×n₄×n₃ such that:

Z(i,j,:)=Σ_(s∈n) ₂ X(i,s,:)⊗Y(s,j,:)  (2)

for i∈[n₁] and j∈[n₄], where each [n] corresponds to an index set {1, 2,. . . , n}, and X(i,s,:) and Y(s,j,:) each correspond to a tube. For thetensor X, its transpose X^(†)∈

^(n) ² ^(×n) ¹ ^(×n) ³ is defined as:

(X ^(†))^((k))=(X ^((k)))^(H)  (3)

where k∈[n₃] and (X^((k)))^(H) denotes the Hermitian (conjugate)transpose of X^((k)). If X is L-diagonal, its (frontal) slicescorrespond to diagonal matrices and X⊗X^(†)=X^(†)⊗X=1, where I∈

^(n×n×n) ³ is L-diagonal with e's on the diagonals. That is, the tubeI(i,s,:)=L⁻¹(1″) for i∈[n], and all other tubes are 0″, where 1″, 0″∈

^(1×1×n) ³ denote tubes with all entries equal to 1 or 0, respectively.The L-SVD of a tensor T∈

^(n) ¹ ^(×n) ² ^(×n) ³ is defined as:

T=U⊗S⊗V ^(†)  (4)

where U∈

^(n) ¹ ^(×n) ² ^(×n) ³ and V∈

^(n) ² ^(×n) ² ^(×n) ³ are L-orthogonal tensors and S∈

^(n) ¹ ^(×n) ² ^(×n) ³ is an L diagonal tensor. Then, the L-tubal rankof T is defined as the number of non-zero tubes of S. Lastly, the tensornuclear norm of the tensor X is defined as:

∥X∥ _(TNN)=Σ_(k∈[n) ₃ _(]) ∥{tilde over (X)} ^((k))∥.  (5)

where ∥⋅∥ denotes the matrix nuclear-norm and {tilde over (X)}^((k)) isa frequency domain representation obtained by taking a Fourier transformalong the third-dimension of X.

Given the above-definitions, the video frame synthesis task can beformally modeled as a tensor completion problem with random missing(frontal) slices. For example, consider a video tensor X∈

^(n) ¹ ^(×n) ² ^(×n) ³ . Let Ω⊆[n₃] be the set of observed video frames.An observation tensor Φ∈

^(n) ¹ ^(×n) ² ^(×n) ³ can be defined having k-th (frontal) slicesΦ(:,:,k) as follows:

$\begin{matrix}{{\Phi\left( {:{,{:{,\ k}}}} \right)} = \left\{ \begin{matrix}{1^{\prime},} & {k \in \Omega} \\{0^{\prime},} & {otherwise}\end{matrix} \right.} & (6)\end{matrix}$

where 1′ denotes an n₁×n₂ matrix of ones and 0′ denotes an n₁×n₂ matrixof zeroes. Then, the observed video frames of a video corresponding to atensor T∈

^(n) ¹ ^(×n) ² ^(×n) ³ can be expressed as Y=Φ⊙T and the k-th (frontal)slices of Y(:,:,k) can be defined as:

$\begin{matrix}{{Y\left( {:{,{:{,\ k}}}} \right)} = \left\{ \begin{matrix}{T^{(k)},} & {k \in \Omega} \\{0,} & {otherwise}\end{matrix} \right.} & (7)\end{matrix}$

where ⊙ again denotes the Hadamard (element-wise) product.

The tensor completion problem can generally be solved since compressivesensing (CS) theory shows that a signal sampled at a rate lower than theNyquist rate can be reconstructed, and an iterative shrinkagethresholding algorithm (ISTA) can be employed to solve CS problems withdense matrix data.

With the foregoing in mind, as shown, the TTNet 120 can include aplurality of phases 122-1 through 122-N. Each of the phases 122-1through 122-N has the same architecture but different parameter values.

The input video data 110 is converted into a tensor X⁰, which isreceived by phase 1 122-1. A tensor output at phase 1 122-1, X¹ is sentto a next phase of the TTNet 120. The N-th phase of the TTNet receives atensor output by the preceding phase of the TTNet 120, X^(N−1), andoutputs a tensor X^(N). The tensor X^(N) corresponds to a synthesizedresult that would be compared with the ground truth to calculatetraining loss for convergence acceleration. A recovered result 130having a plurality of frames including frame 132 is obtained. Therecovered result 130 corresponds to the input video data with the one ormore missing frames included.

In view of the above definitions, the video frame synthesis task can becast as completing the tensor T from the observed frontal slices indexedby Ω. Thus, the objective becomes finding a video tensor X with anL-tubal rank≤r such that Φ⊗X=Y. More specifically, the tensor completionproblem is provided as follows:

$\begin{matrix}{\begin{matrix}{argmin} \\{X \in {\mathbb{R}}^{n_{1} \times n_{2} \times n_{3}}}\end{matrix}\left\{ {{\frac{1}{2}{{{\Phi \odot X} - Y}}_{F}^{2}} + {\lambda{X}_{TNN}}} \right\}} & (8)\end{matrix}$

where ∥T∥_(F) ² for some tensor T is defined as √{square root over(Σ_(i=1) ^(n) ¹ Σ_(j=1) ^(n) ² Σ_(k=1) ^(n) ³ |T(i,j,k)|²)}, and λ is aconstant parameter. The term λ∥X∥_(TNN) is used to impose sparsity. Aswill be described in further detail below with reference to FIG. 2, eachof the phase 122-1 through 122-N includes a plurality of components ormodules designed to implement an inference iteration of a tensor ISTAalgorithm used to solve the tensor completion problem. For t∈[1,N], thetensor ISTA algorithm performed at phase t is proposed as follows:

$\begin{matrix}{\begin{matrix}{argmin} \\{X^{t} \in {\mathbb{R}}^{n_{1} \times n_{2} \times n_{3}}}\end{matrix}\left\{ {{\frac{1}{2}{{X - R^{t}}}_{F}^{2}} + {\lambda{X}_{TNN}}} \right\}} & (9)\end{matrix}$R ^(t) =X ^(t−1)−ρΦ⊙(⊙X ^(t−1) −Y)  (10)

where ρ is a step size and R^(t) is an intermediate synthesis result inan original domain.

As will be described in further detail below with reference to FIG. 2, agiven one of the plurality of phases 122-1 through 122-N can include anupdate module, a transform module, a soft-thresholding module, and aninverse transform module. The update module calculates the gradient of∥Φ⊗X^(†)−Y∥_(F) ² by using equation (9) to update the intermediatesynthesis result in the original domain R^(t), the transform moduletransforms the intermediate synthesis result into a transformedintermediate synthesis result in a transform domain (R ^(t)), thesoft-thresholding module applies a soft-thresholding operation on each(frontal) slice of R ^(t) to generate synthesized video data in thetransform domain (X ^(†,(k))), and the inverse transform moduletransforms the synthesized video data in the transform domain back tothe original domain.

A loss function including two loss terms can be used. One loss termevaluates the accuracy of the synthesized video frames and the otherloss term imposes the sparsity of video frame tensors in the transformdomain. For example, a loss function L including an accuracy loss termL_(accuracy) and a sparsity loss term L_(sparsity) can be defined as:

L=αL _(accuracy) +βL _(sparsity)  (11)

where α and β are parameters that balance the accuracy and sparsity lossterms. In an illustrative embodiment, α and β can be set to 1 and 0.1 bydefault, respectively. For example, L_(accuracy) can be defined as∥X^(N)−X∥_(F) ² and L_(sparsity) can be defined as

${\frac{1}{N}{\sum\limits_{t = 1}^{N}\;{X^{N}}_{TNN}}},$

such that

$\begin{matrix}{L = {{\alpha{{X^{N} - X}}_{F}^{2}} + {\frac{\beta}{N}{\sum\limits_{t = 1}^{N}{X^{N}}_{TNN}}}}} & (12)\end{matrix}$

The tensor ISTA algorithm in accordance with the embodiments describedherein provides improvements over conventional ISTA algorithms thattransform the video data into matrix representations which could losespatiotemporal information.

Referring now to FIG. 2, a block/flow diagram is provided illustratingan overview of a phase of a transform-based tensor neural network(TTNet) 200.

As shown, an input tensor generated at the t−1-th phase, X^(t−1) 210,where X^(t−1)∈

^(n) ¹ ^(×n) ² ^(×n) ³ , is received by a phase 220 of the TTNet 200.The phase 220 corresponds to a given one of a plurality of N phases ofthe TTNet, as described above with reference to FIG. 1. Morespecifically, the phase 220 is a t-th phase configured to generate atensor output X^(t) 230, where X^(t)∈

^(n) ¹ ^(×n) ² ^(×n) ³ . Accordingly, if, t=N, then X^(t) 230 equalsX^(N) corresponds to a final tensor output of the TTNet.

The phase 220 corresponds to one inference iteration of a tensoriterative shrinkage thresholding algorithm (ISTA) implemented by theTTNet 200 used to generate the final output tensor X^(N). Morespecifically, phase 220 includes a plurality of modules configured toimplement the inference iteration of the tensor ISTA algorithm,including an update module 222, a transform module 224, asoft-thresholding module 226, and an inverse transform module 228.

The update module 222 is configured to generate an updated intermediatesynthesis result in an original domain as described in further detailabove with reference to FIG. 1. More specifically, the update module 222can calculate the gradient of ∥Φ⊗X^(t)−Y∥_(F) ² by using equation (9),where R^(t)=X^(t−1)−ρΦ⊙(⊙X^(t−1)−Y), to generate the updatedintermediate synthesis result in the original domain.

The aim of the TTNet 200 is to find a solution to the tensor completionproblem in a learned transform domain, and then transform the solutionback to the original domain. To achieve this, the transform module 224is configured transform the updated intermediate synthesis result into atransformed intermediate result in a transform domain. Further detailsregarding the transform module will now be described below withreference to FIG. 3.

With reference to FIG. 3, a block/flow diagram is provided illustratingan exemplary transform-based tensor neural network (TTNet) 300. Asshown, the TTNet 300 includes the update module 222 configured toreceive the tensor input 210, the transform module 224, and thesoft-thresholding module 226 described above with reference to FIG. 2.

As further shown, in this illustrative example, the transform module 224includes a plurality of components including a first convolution (Cony)component 310-1, an activation function (AF) component 320 and a secondCony component 310-2. In one embodiment, the first and second Conycomponents 310-1 and 310-2 implement two-dimensional (2D) multi-channelconvolutions with different kernels, and the AF component 320 includes arectified linear unit (ReLU) that incorporates nonlinearity.

The first Cony component 310-1 receives the updated intermediatesynthesis result from the update module 222 to generate Conv(R^(t)).Then, the AF component 320 applies the activation function toConv(R^(t)). For example, if the AF component 320 includes a ReLU, thenthe application of the activation function to Conv(R^(t)) generatesReLU(Conv(R^(t))). Then, the second Cony component 310-2 applies asecond convolution to ReLU(Conv(R^(t))) to generateConv(ReLU(Conv(R^(t))))=F(R^(t))=R ^(t), which corresponds to thetransformed intermediate synthesis result.

Referring back to FIG. 2, after the transformed intermediate synthesisresult, F(R^(t))=R ^(t), has been obtained, the soft-thresholding module226 is configured to apply soft-thresholding based on the transformedintermediate synthesis result to generate synthesized video data in thetransform domain. More specifically, the synthesized video data in thetransform domain corresponds to a tensor. In one embodiment, thesoft-thresholding module 226 is configured to apply thesoft-thresholding by applying a soft-thresholding operation on each(frontal) slice of R ^(t) separately to generate the synthesized videodata in the transform domain. Further details regarding thesoft-thresholding module 226 will now be described below with referenceto FIG. 4.

With reference to FIG. 4, a block/flow diagram is provided illustratingan exemplary transform-based tensor neural network (TTNet) 400. Asshown, the TTNet 400 includes the transform module 224, thesoft-thresholding module 226, and the inverse transform module 228described above with reference to FIG. 2.

As further shown, in this illustrative example, the soft-thresholdingmodule 226 includes a plurality of soft-thresholding components 410-1through 410-Q configured to perform parallel soft-thresholdingoperations, where Q equals n₃, as described above with reference toFIG. 1. More specifically, each of the plurality of soft-thresholdingcomponents 410-1 through 410-Q is configured to receive a respective oneof F(R^(t))⁽¹⁾ through F(R^(t))^((Q)) as input and generate a respective(matrix) output. After the parallel soft-thresholding operations areperformed, the outputs of the plurality of soft-thresholding components410-1 through 410-Q are stacked together to form a tensor, F(X^(t))=X^(t), corresponding to synthesized video data in the transform domain.For example:

F(X ^(t))^((k))=soft(F(R ^(t))^((k)),λ)  (13)

where soft(⋅) is the element-wise soft-thresholding operation defined asfollows:

$\begin{matrix}{{{soft}\left( {Y_{ij},\lambda} \right)} = \left\{ \begin{matrix}{Y_{ij} + \lambda} & {{{if}\mspace{14mu} Y_{ij}} \leq {- \lambda}} \\0 & {{{if}\mspace{14mu}{Y_{ij}}} \leq \lambda} \\Y_{ij} & {{{if} Y_{ij}} \geq {- \lambda}}\end{matrix} \right.} & (14)\end{matrix}$

and where k∈[n₃] and A is a constant.

Referring back to FIG. 2, after the synthesized video data in thetransform domain has been generated, the inverse transform module 228 isconfigured to transform the synthesized video data in the transformdomain back into the original domain to generate synthesized video datain the original domain, corresponding to X^(t) 230. Further detailsregarding the inverse transform module 228 will now be described belowwith reference to FIG. 5.

With reference to FIG. 5, a block/flow diagram is provided illustratingan exemplary transform-based tensor neural network (TTNet) 500. Asshown, the TTNet 500 includes the soft-thresholding module 226 and theinverse transform module 228 configured to generate the tensor output230 described above with reference to FIG. 2.

As further shown, in this illustrative example, the inverse transformmodule 228 includes a plurality of components including a firstconvolution (Cony) component 510-1, an activation function (AF)component 520 and a second Cony component 510-2. In one embodiment, thefirst and second Cony components 510-1 and 510-2 implementtwo-dimensional (2D) multi-channel convolutions with different kernels,and the AF component 520 includes a rectified linear unit (ReLU) thatincorporates nonlinearity. The network structure of the inversetransform module 228 is similar to that of the transform module 224,except that their Cony components have different kernel parameters.

The first Cony component 510-1 receives the synthesized video data inthe transform domain from the soft-thresholding module 222 to generateConv(X ^(t)). Then, the AF component 520 applies the activation functionto Conv(X ^(t)). For example, if the AF component 520 includes arectified linear unit (ReLU), then the application of the activationfunction to Conv(X ^(t)) generates ReLU(Conv(X ^(t))). Then, the secondCony component 510-2 applies a second convolution to ReLU(Conv(X ^(t)))to generate Conv(ReLU(Conv(X ^(t))))=F⁻¹(X^(t))=F⁻¹F((X^(t)))=X^(t) 230.

Referring back to FIG. 2, a loss function can be used as described abovewith reference to FIG. 1. More specifically, the loss function can be alinear combination of a first part that evaluates the accuracy ofsynthesized video frames and a second part that imposes sparsity.Further details regarding the loss function are described above withreference to FIG. 1.

Exemplary pseudocode 600 describing the ISTA algorithm iterated from t=0to t=N to generate a final tensor output X^(N) is illustrated withreference to FIG. 6.

Referring now to FIG. 7, a block/flow diagram is provided showing asystem/method 700 for implementing video frame synthesis using atransform-based tensor neural network (TTNet). More specifically, thesystem/method 700 can be used to perform interpolation and/or predictionof video frames. For example, the system/method 700 can be used toimprove the peak signal-to-noise (PSNR) of interpolation and prediction(e.g., 4.13 dB and 4.16 dB), reduce motion blurring, and avoidartifacts.

At block 710, input video data including one or more missing frames isreceived.

At block 720, the input video data is converted into an input tensor.

At block 730, output video data including one or more synthesized framescorresponding to the one or more missing frames is generated throughtensor completion based on the input tensor by using a transform-basedtensor neural network (TTNet) including a plurality of phasesimplementing a tensor iterative shrinkage thresholding algorithm (ISTA).More specifically, each of the plurality of phases can include aplurality of modules configured to implement one inference iteration ofthe tensor ISTA. Further details regarding implementing an inferenceiteration of the tensor ISTA will be described in further detail belowwith reference to FIG. 8.

At block 740, a loss function is obtained based on the output videodata. In one embodiment, the loss function includes a first part thatevaluates the accuracy of the one or more synthesized frames, and asecond part that imposes sparsity. More specifically, the loss functionmay be a linear combination of the first and second parts. For example,the loss function can be defined by equation (11), as described abovewith reference to FIG. 1.

Further details regarding block 710-750 of FIG. 7 are described abovewith reference to FIGS. 1-6.

Referring now to FIG. 8, a block/flow diagram is provided illustrating asystem/method 800 for implementing an inference iteration of a tensoriterative shrinkage thresholding algorithm (ISTA).

At block 810, for a given one of a plurality of phases of atransform-based tensor neural network (TTNet), an intermediate synthesisresult in an original domain is updated. Further details regarding block810 are described above with reference to FIG. 2.

At block 820, the intermediate synthesis result in the original domainis transformed into a transformed intermediate result in a transformdomain. More specifically, transforming the intermediate synthesisresult in the original domain into the transformed intermediate resultcan include applying a first convolution to the intermediate synthesisresult in the original domain, applying an activation function to theoutput of the first convolution, and applying a second convolution tothe output of the activation function. In one embodiment, the first andsecond convolutions include two-dimensional (2D) multi-channelconvolutions with different kernels, and the activation function is arectified linear unit (ReLU) that incorporates nonlinearity. Furtherdetails regarding block 820 are described above with reference to FIGS.2 and 3.

At block 830, soft-thresholding is applied based on the transformedintermediate synthesis result to generate synthesized video data in thetransform domain. More specifically, the synthesized video data in thetransform domain corresponds to a tensor. In one embodiment, applyingthe soft-thresholding based on the transformed intermediate synthesisresult to generate the synthesized video data in the transform domainincludes applying a plurality of soft-thresholding operations to each(frontal) slice of the transformed intermediate synthesis result, andstacking outputs of each of the plurality of soft-threshold operationsto form the synthesized video data in the transform domain. Furtherdetails regarding block 830 are described above with reference to FIGS.2 and 4.

At block 840, the synthesized video data in the transform domain istransformed back to the original domain using an inverse transformation.More specifically, transforming the synthesized video data in thetransform domain back to the original domain using an inversetransformation can include applying a third convolution to thesynthesized video data in the transform domain, applying a secondactivation function to the output of the third convolution, and applyinga fourth convolution to the output of the third activation function. Inone embodiment, the third and fourth convolutions include 2Dmulti-channel convolutions with different kernels, and the activationfunction is a ReLU that incorporates nonlinearity. The third and fourthconvolutions can have different kernel parameters than the first andsecond convolutions of block 820. Further details regarding block 840are described above with reference to FIGS. 2 and 5.

Further details regarding the ISTA algorithm are described above withreference to FIG. 6.

Referring now to FIG. 9, a block diagram is provided showing anexemplary processing system 900, in accordance with an embodiment of thepresent invention. The processing system 900 includes a set ofprocessing units (e.g., CPUs) 901, a set of GPUs 902, a set of memorydevices 903, a set of communication devices 904, and set of peripherals905. The CPUs 901 can be single or multi-core CPUs. The GPUs 902 can besingle or multi-core GPUs. The one or more memory devices 903 caninclude caches, RAMs, ROMs, and other memories (flash, optical,magnetic, etc.). The communication devices 904 can include wirelessand/or wired communication devices (e.g., network (e.g., WIFI, etc.)adapters, etc.). The peripherals 905 can include a display device, auser input device, a printer, an imaging device, and so forth. Elementsof processing system 900 are connected by one or more buses or networks(collectively denoted by the figure reference numeral 910).

In an embodiment, memory devices 903 can store specially programmedsoftware modules configured to implement various aspects of the presentinvention. In an embodiment, special purpose hardware (e.g., ApplicationSpecific Integrated Circuits, Field Programmable Gate Arrays (FPGAs),and so forth) can be used to implement various aspects of the presentinvention.

In an embodiment, memory devices 903 store program code 906 forimplementing a transform-based tensor neural network (TTNet). Asdescribed in further detail above, the TTNet can be used to performvideo frame synthesis (e.g., video frame interpolation and/or videoframe prediction) in an improved manner.

Of course, the processing system 900 may also include other elements(not shown), as readily contemplated by one of skill in the art, as wellas omit certain elements. For example, various other input devicesand/or output devices can be included in processing system 900,depending upon the particular implementation of the same, as readilyunderstood by one of ordinary skill in the art. For example, varioustypes of wireless and/or wired input and/or output

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 10, illustrative cloud computing environment 1050is depicted. As shown, cloud computing environment 1050 includes one ormore cloud computing nodes 1010 with which local computing devices usedby cloud consumers, such as, for example, personal digital assistant(PDA) or cellular telephone 1054A, desktop computer 1054B, laptopcomputer 1054C, and/or automobile computer system 1054N may communicate.Nodes 1010 may communicate with one another. They may be grouped (notshown) physically or virtually, in one or more networks, such asPrivate, Community, Public, or Hybrid clouds as described hereinabove,or a combination thereof. This allows cloud computing environment 1050to offer infrastructure, platforms and/or software as services for whicha cloud consumer does not need to maintain resources on a localcomputing device. It is understood that the types of computing devices1054A-N shown in FIG. 2 are intended to be illustrative only and thatcomputing nodes 1010 and cloud computing environment 1050 cancommunicate with any type of computerized device over any type ofnetwork and/or network addressable connection (e.g., using a webbrowser).

Referring now to FIG. 11, a set of functional abstraction layersprovided by cloud computing environment 1050 (FIG. 10) is shown. Itshould be understood in advance that the components, layers, andfunctions shown in FIG. 11 are intended to be illustrative only andembodiments of the invention are not limited thereto. As depicted, thefollowing layers and corresponding functions are provided:

Hardware and software layer 1160 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 1161;RISC (Reduced Instruction Set Computer) architecture based servers 1162;servers 1163; blade servers 1164; storage devices 1165; and networks andnetworking components 1166. In some embodiments, software componentsinclude network application server software 1167 and database software1168.

Virtualization layer 1170 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers1171; virtual storage 1172; virtual networks 1173, including virtualprivate networks; virtual applications and operating systems 1174; andvirtual clients 1175.

In one example, management layer 1180 may provide the functionsdescribed below. Resource provisioning 1181 provides dynamic procurementof computing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 1182provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 1183 provides access to the cloud computing environment forconsumers and system administrators. Service level management 1184provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 1185 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 1190 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 1191; software development and lifecycle management 1192;virtual classroom education delivery 1193; data analytics processing1194; transaction processing 1195; and video synthesis 1196.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

Reference in the specification to “one embodiment” or “an embodiment” ofthe present invention, as well as other variations thereof, means that aparticular feature, structure, characteristic, and so forth described inconnection with the embodiment is included in at least one embodiment ofthe present invention. Thus, the appearances of the phrase “in oneembodiment” or “in an embodiment”, as well any other variations,appearing in various places throughout the specification are notnecessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following “/”,“and/or”, and “at least one of”, for example, in the cases of “A/B”, “Aand/or B” and “at least one of A and B”, is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of both options (A andB). As a further example, in the cases of “A, B, and/or C” and “at leastone of A, B, and C”, such phrasing is intended to encompass theselection of the first listed option (A) only, or the selection of thesecond listed option (B) only, or the selection of the third listedoption (C) only, or the selection of the first and the second listedoptions (A and B) only, or the selection of the first and third listedoptions (A and C) only, or the selection of the second and third listedoptions (B and C) only, or the selection of all three options (A and Band C). This may be extended, as readily apparent by one of ordinaryskill in this and related arts, for as many items listed.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

As employed herein, the term “hardware processor subsystem” or “hardwareprocessor” can refer to a processor, memory, software or combinationsthereof that cooperate to perform one or more specific tasks. In usefulembodiments, the hardware processor subsystem can include one or moredata processing elements (e.g., logic circuits, processing circuits,instruction execution devices, etc.). The one or more data processingelements can be included in a central processing unit, a graphicsprocessing unit, and/or a separate processor- or computing element-basedcontroller (e.g., logic gates, etc.). The hardware processor subsystemcan include one or more on-board memories (e.g., caches, dedicatedmemory arrays, read only memory, etc.). In some embodiments, thehardware processor subsystem can include one or more memories that canbe on or off board or that can be dedicated for use by the hardwareprocessor subsystem (e.g., ROM, RAM, basic input/output system (BIOS),etc.).

In some embodiments, the hardware processor subsystem can include andexecute one or more software elements. The one or more software elementscan include an operating system and/or one or more applications and/orspecific code to achieve a specified result.

In other embodiments, the hardware processor subsystem can includededicated, specialized circuitry that performs one or more electronicprocessing functions to achieve a specified result. Such circuitry caninclude one or more application-specific integrated circuits (ASICs),FPGAs, and/or PLAs.

These and other variations of a hardware processor subsystem are alsocontemplated in accordance with embodiments of the present invention.

Having described preferred embodiments of video synthesis (which areintended to be illustrative and not limiting), it is noted thatmodifications and variations can be made by persons skilled in the artin light of the above teachings. It is therefore to be understood thatchanges may be made in the particular embodiments disclosed which arewithin the scope of the invention as outlined by the appended claims.Having thus described aspects of the invention, with the details andparticularity required by the patent laws, what is claimed and desiredprotected by Letters Patent is set forth in the appended claims.

1. A method for implementing video frame synthesis using a tensor neuralnetwork, comprising: receiving input video data including one or moremissing frames; converting the input video data into an input tensor;generating, through tensor completion based on the input tensor, outputvideo data including one or more synthesized frames corresponding to theone or more missing frames by using a transform-based tensor neuralnetwork (TTNet) including a plurality of phases implementing a tensoriterative shrinkage thresholding algorithm (ISTA); and obtaining a lossfunction based on the output video data.
 2. The method of claim 1,wherein generating the output video data further includes: for a givenone of the plurality of phases of the TTNet, transforming anintermediate synthesis result in an original domain into a transformedintermediate synthesis result in a transform domain; applyingsoft-thresholding based on the transformed intermediate synthesis resultto generate synthesized video data in the transform domain; andtransforming the synthesized video data in the transform domain back tothe original domain using an inverse transformation.
 3. The method ofclaim 2, further comprising updating the intermediate result in theoriginal domain prior to transforming the intermediate result in theoriginal domain, wherein the intermediate synthesis result is definedbased in part on a video tensor received by the given one of theplurality of phases and an observation tensor.
 4. The method of claim 2,wherein transforming the intermediate synthesis result in the originaldomain into the transformed intermediate synthesis result furtherincludes applying a first convolution to the intermediate synthesisresult, applying an activation function to an output of the firstconvolutional network, and applying a second convolution to an output ofthe activation function to generate the transformation.
 5. The method ofclaim 4, wherein the first and second convolutions includetwo-dimensional (2D) multi-channel convolutions with different kernels,and the activation function is a rectified linear unit (ReLU).
 6. Themethod of claim 2, wherein applying soft-thresholding based on thetransformed intermediate synthesis result to generate the synthesizedvideo data in the transform domain further includes applying a pluralityof soft-thresholding operations in parallel to each frontal slice of thetransformed intermediate synthesis result, and stacking outputs of theplurality of soft-thresholding operations to generate the synthesizedvideo data in the transform domain.
 7. The method of claim 2, whereintransforming the synthesized video data in the transform domain back tothe original domain using an inverse transformation further includesapplying a first convolution to the synthesized video data in thetransform domain, applying an activation function to an output of thefirst convolution, and applying a second convolution to an output of theactivation function.
 8. The method of claim 7, wherein the first andsecond convolutions include 2D multi-channel convolutions with differentkernels, and the activation function is a ReLU.
 9. The method of claim1, where the loss function includes a first part that evaluates anaccuracy of the one or more synthesized frames, and a second part thatimposes sparsity.
 10. A computer readable storage medium comprising acomputer readable program for implementing video frame synthesis using atensor neural network, wherein the computer readable program whenexecuted on a computer causes the computer to perform a methodincluding: receiving input video data including one or more missingframes; converting the input video data into an input tensor;generating, through tensor completion based on the input tensor, outputvideo data including one or more synthesized frames corresponding to theone or more missing frames by using a transform-based tensor neuralnetwork (TTNet) including a plurality of phases implementing a tensoriterative shrinkage thresholding algorithm (ISTA); and obtaining a lossfunction based on the output video data.
 11. The computer readablestorage medium of claim 10, wherein generating the output video datafurther includes: for a given one of the plurality of phases of theTTNet, transforming an intermediate synthesis result in an originaldomain into a transformed intermediate synthesis result in a transformdomain; applying soft-thresholding based on the transformed intermediatesynthesis result to generate synthesized video data in the transformdomain; and transforming the synthesized video data in the transformdomain back to the original domain using an inverse transformation. 12.The computer readable storage medium of claim 11, wherein the methodfurther includes updating the intermediate result in the original domainprior to transforming the intermediate result in the original domain,and wherein the intermediate synthesis result is defined based in parton a video tensor received by the given one of the plurality of phasesand an observation tensor.
 13. The computer readable storage medium ofclaim 11, wherein: transforming the intermediate synthesis result in theoriginal domain into the transformed intermediate synthesis resultfurther includes applying a first convolution to the intermediatesynthesis result, applying a first rectified linear unit (ReLU) to anoutput of the first convolution, and applying a second convolution to anoutput of the first ReLU to generate the transformation, wherein thefirst and second convolutions include two-dimensional (2D) multi-channelconvolutions with different kernels; transforming the synthesized videodata in the transform domain back to the original domain using aninverse transformation further includes applying a third convolution tothe synthesized video data in the transform domain, applying a secondReLU to an output of the third convolution, and applying a fourthconvolution to an output of the second ReLU; and the first, second,third and fourth convolutions include 2D multi-channel convolutions withdifferent kernels.
 14. The computer readable storage medium of claim 11,wherein applying soft-thresholding based on the transformed intermediatesynthesis result to generate the synthesized video data in the transformdomain further includes applying a plurality of soft-thresholdingoperations in parallel to each frontal slice of the transformedintermediate synthesis result, and stacking outputs of the plurality ofsoft-thresholding operations to generate the synthesized video data inthe transform domain.
 15. The computer readable storage medium of claim11, wherein the loss function includes a first part that evaluates anaccuracy of the one or more synthesized frames, and a second part thatimposes sparsity.
 16. A system for implementing video frame synthesisusing a tensor neural network, comprising: a memory configured to storeprogram code; and at least one processor device operatively coupled tothe memory and configured to execute program code stored on the memoryto: receive input video data including one or more missing frames;convert the input video data into an input tensor; generate, throughtensor completion based on the input tensor, output video data includingone or more synthesized frames corresponding to the one or more missingframes by using a transform-based tensor neural network (TTNet)including a plurality of phases implementing a tensor iterativeshrinkage thresholding algorithm (ISTA); and obtain a loss functionbased on the output video data.
 17. The system of claim 16, wherein theat least one processor device is further configured to generate theoutput video data by: for a given one of the plurality of phases of theTTNet, transforming an intermediate synthesis result in the originaldomain into a transformed intermediate synthesis result in a transformdomain; applying soft-thresholding based on the transformed intermediatesynthesis result to generate synthesized video data in the transformdomain; and transforming the synthesized video data in the transformdomain back to the original domain using an inverse transformation. 18.The system of claim 17, wherein: the at least one processor device isfurther configured to transform the intermediate synthesis result in theoriginal domain into the transformed intermediate synthesis result byapplying a first convolution to the intermediate synthesis result,applying a first rectified linear unit (ReLU) to an output of the firstconvolution, and applying a second convolution to an output of the firstReLU to generate the transformation; the at least one processor deviceis further configured to transform the synthesized video data in thetransform domain back to the original domain using an inversetransformation by applying a third convolution to the synthesized videodata in the transform domain, applying a second ReLU to an output of thethird convolution, and applying a fourth convolution to an output of thesecond ReLU; and the first, second, third and fourth convolutionsinclude 2D multi-channel convolutions with different kernels.
 19. Thesystem of claim 17, wherein the at least one processor device is furtherconfigured to apply soft-thresholding based on the transformedintermediate synthesis result to generate the synthesized video data inthe transform domain by applying a plurality of soft-thresholdingoperations in parallel to each frontal slice of the transformedintermediate synthesis result, and stacking outputs of the plurality ofsoft-thresholding operations to generate the synthesized video data inthe transform domain.
 20. The system of claim 16, wherein the lossfunction includes a first part that evaluates an accuracy of the one ormore synthesized frames, and a second part that imposes sparsity.